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APPARATUS, SYSTEMS AND METHODS FOR 
C ONTROLLING GRA PHICS AND VID EO DATA IN MULTIMEDIA DATA 
" PROCESSING AND DISPLAY S YSTEMS 



TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to 
multimedia processing and display systems and in 
particular to apparatus, systems and methods for 
controlling graphics and video data overlay in multimedia 
processing and display systems. 



CROSS-REFERENCE TO RELATED APPLICATIONS 

The following copending and coassigned United States 
patent applications contain related information and are 

10 incorporated herein by reference: 

United States Patent Application Serial No. 
08/098,846 (Attorney's Docket No. P3510-P11US) , entitled 
"System And Method For The Mixing Of Graphics And Video 
Signals," and filed July 29, 1993; and 

15 United States Patent Application Serial No. 

08/223,845 (Attorney's Docket No. P3510-P21US) , entitled 
"Apparatus, Systems And Methods For Processing Video Data 
In Conjunction With A Multi-Format Frame Buffer," and 
filed April 6, 1994. 
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BACKGROUND OF THE INVENTION 

As multimedia information processing systems 
increase in popularity, system designers must consider 
new techniques for controlling the processing and display 
5 of data simultaneously generated by multiple sources. In 
particular, there has been substantial demand for 
processing systems which have the capability of 
concurrently displaying both video and graphics data on 
a single display screen. The development of such systems 

10 presents a number of design challenges, not only because 
the format differences between graphics and video data 
must be accounted for, but also because of end user 
driven requirements that these systems allow for flexible 
manipulation of the data on the display screen. 

15 One particular technique for simultaneously 

displaying video and graphics data on a single display 
screen involves the generation of "windows." In this 
case, a stream of data from a selected source is used to 
generate a display within a particular region or "window" 

20 of the display screen to the exclusion of any nonselected 
data streams defining a display or part of a display 
corresponding to the same region of the screen. The 
selected data stream generating the display window 
"overlays" or "occludes" the data from the nonselected 

25 data streams which lie "behind" the displayed data. In 
one instance, the overall content and appearance of the 
display screen is defined by graphics data and one or 
more "video windows" generated by data from a video 
source occlude a corresponding region of that graphics 

30 data. In other instances, a video display or window may 
be occluded or overlaid by graphics data or even another 
video window. 
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In the multimedia environment, the "windowing" 
described above yields substantial advantages. Among 
other things, the user can typically change the size and 
location on the display screen of a given window to 
flexibly manipulate the content and appearance of the 
data being displayed. For example, in the case of 
combined graphics and video, the user can advantageously 
create custom composite visual displays by combining 
multiple video and graphics data streams in windowing 
environment - 

In order to efficiently control windows in a 
multimedia environment efficient frame buffer management 
is required. Specifically, a frame buffer control scheme 
must be developed which allows for the efficient storage 
and retrieval of multiple types of data, such as video 
data and graphics data. To be cost competitive as well 
as functionally efficient, such a scheme should minimize 
the number of memory devices and the amount of control 
circuitry required and should insure that data flow to 
the display is subjected to minimal delay notwithstanding 
data type. 

One of the major difficulties in managing video in 
a combined video and graphics windowing environment 
results from the fact that the video data being received 
and displayed are constantly being updated, typically at 
a rate of thirty frames per second. In contrast, the 
graphics data are normally generated once to define the 
graphics display and then remain static until the system 
CPU change that graphics display. Thus, the occlusion 
(overlay) of video data with graphics data requires that 
the static graphics data "in front of" the video data not 
be destroyed each time the video window is updated. A 
second concern with windowing systems operating on both 
video and graphics data is the formatting differences 
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between the video and graphics data themselves since 
video is typically digitized into a YUV color space while 
graphics is digitized into an RGB color space. Hence, 
any combination video and graphics windowing system must 
have the capability of efficiently handling data within 
both the YUV and RGB formats. 

Thus, due to the advantages of windowing, the need 
has arisen for efficient and cost effective windowing 
control circuitry. Such windowing circuitry should allow 
for the simultaneous processing of data received from 
multiple sources and in multiple formats. In particular, 
such windowing control circuitry should be capable of 
efficiently and inexpensively controlling the occlusion 
and/or overlay of video and graphics data in a windowing 
environment . 
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SUMMARY OF THE INVENTION 

The principles of the present invention in general 
provide for the flexible control of graphics and video 
data in a display control environment. In particular, an 
5 entire frame of video data, graphics data, or a 
combination of both, may be stored in on-screen memory 
and rastered out with the generation of the corresponding 
display screen. A window of graphics or video data can 
then be stored in off-screen memory and retrieved when 

10 the raster scan generating the display reaches the 
desired position on the display for the video window. 
The window of data from off-screen memory can then be 
overlayed over the data being rastered out of the 
on-screen memory under one of three conditions. In a 

15 first mode, pixels from the off-screen memory are 
rastered only when the raster scan has reached the 
position on the display selected for the window. In a 
second mode, a window of data is rastered from the 
off-screen memory when the display raster scan has 

20 reached the display window position and graphics data 
being rastered from the on-screen memory matches a color 
key. In a third mode, the window data is rastered out of 
the off-screen memory when the data being output from the 
on-screen memory matches the color key, notwithstanding 

25 the position of the raster scan. 

According to a first embodiment of the present 
invention, a graphics and video controller is provided 
which includes a dual aperture interface, each word 
associated with an address to a selected one of on-screen 

30 and off-screen areas of an associated unified frame 
buffer as either graphics or video pixel data. Circuitry 
is provided for writing a word of the pixel data received 
by the interface to a one of the on-screen and off-screen 
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memory areas corresponding to the address associated with 



selectively retrieving graphics and video data from the 
on-screen and off-screen memory areas, A first pipeline 
is provided for processing graphics data retrieved from 
the frame buffer and a second pipeline is provided for 
video processing data retrieved from the frame buffer. 

According to a second embodiment of the present 
invention, a controller is provided which includes a dual 
aperture port for receiving video and graphics data, each 
word of the data received with an address associated 
directing the word to be processed as either graphics or 
video data and off-screen memory spaces of a frame 
buffer. A second port is included for receiving 
real-time video data. Circuitry is provided for 
generating an address associated with a selected one of 
the memory spaces for each word of received real-time 
video data. Circuitry is included for writing 

selectively the words into the on-screen and off-screen 
memory spaces of the frame buffer. Circuitry is also 
provided for selectively retrieving the words of data 
from the on-screen and off-screen spaces as data is 
rastered for driving a display. A graphics backend 
pipeline processes ones of the graphics words of data 
retrieved from the frame buffer. A video backend 
pipeline is provided for processing ones of the video 
words of data retrieved from the frame buffer, the 
circuitry for retrieving always rastering a stream of 
graphics data from the frame buffer to the graphics 
pipeline and rastering video data to the video backend 
pipeline when a display raster scan reaches a display 
position of a video window. An output selector is 
included for selecting for output between words of data 



the received word. 



Circuitry is also included for 
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output from the graphics backend pipeline and words of 
data output from the video backend pipeline. 
\ j According a third embodiment of the present 

^invention, a display system is provided which includes 
5 first and second parallel backend pipelines. A 
multi-format frame buffer memory is included having 
on-screen and off-screen memories each operable to 
simultaneously store data in graphics and video formats. 
A dual aperture port is provided for receiving both 
10 graphics and video data as directed by an address 
associated with each word of data received. Circuitry 
for writing is included for writing a word of video or 
graphics data into a selected one of the on-screen and 
off-screen areas of the multi-frame buffer. Memory 
15 control circuitry controls the transfer of data between 
the first and second back -end pipelines and the frame 
buffer. The system further includes a display unit and 
overlay control circuitry for selecting for output to the 
display unit between data provided by the first backend 
20 pipeline and data provided by the second backend 
pipeline. 

A fourth embodiment of the present invention 
comprises a display data processing system which includes 
circuitry for writing data into an on-screen space of a 

25 frame buffer and circuitry for writing data into an 
off -screen space of the frame buffer. A video pipeline 
is provided for processing video data output from a 
selected one of the on-screen and off-screen spaces. The 
video pipeline includes a first first-in/first-out memory 

30 for receiving selected pixel data from the selected 
space. The video pipeline also includes a second 
first-in/first-out memory disposed in parallel to the 
first first-in/first-out memory for receiving other 
selected data from the selected space in the frame 
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buffer. An interpolator is provided as part of the video 
pipeline for generating additional data by interpolating 
data output from the first and second first-in/first-out 
memories. A graphics pipeline is disposed in parallel to 
5 the video pipeline for processing graphics data output 
from a selected one of the on-screen and off-screen 
spaces. Finally, an output selector is provided for 
selecting between data output from the video pipeline and 
data output from the graphics pipeline. 

10 The principles of the present invention allow for 

the construction of circuits and systems with substantial 
advantages over the prior art. Among other things, the 
principles of the present invention allow both graphics 
and video data to be stored in a single unified frame 

15 buffer and retrieved therefrom in a number of different 
ways. For example, a combination of graphics and video 
data may be stored in the on-screen memory and simply 
rastered out during screen refresh- In another case, an 
entire screen of graphics or video data may be stored in 

20 the on-screen memory while a window of graphics or video 
data is stored in the off-screen portion of memory. The 
window data can then be rastered out to selectively 
overlay a portion of the data being rastered out of the 
on-screen memory. The overlay may be controlled by 

25 either window display position with a match of the 
on-screen data being rastered out and a color key, or 
both. 

The embodiments of the present invention provide for 
the efficient and inexpensive overlay of video and 
30 graphics data in a windowing environment. In particular, 
the use of color comparison to determine the overlay of 
data in a window region eliminates the need for precise 
x- and y-position data for the location of that window 
and allows for video cropping to be performed. Further, 
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the use of graphics data to control overlay provides 
substantial advantages in that graphics data is less 
subject to the graininess and noise problems often found 
with video data. Further, the user is given total 
5 control of overlay operations when keying on graphics 
data because the graphics data is computer generated, 
whereas the video data is captured data. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, and the advantages thereof, reference is now 
made to the following descriptions taken in conjunction 
5 with thfe accompanying drawings, in which: 

FIGURE 1 is a top level functional block diagram of 
a multi-Tmedia processing and display system embodying the 
principles of the present invention; 

FIGURE 2 is a more detailed functional block diagram 
10 of the /VGA controller depicted in FIGURE 1; 

FIGURE 3 is an expanded functional block diagram of 
portions of the controller of FIGURE 2 with emphasis on 
the overlay control features; 

FIGURE 4A is a detailed functional block diagram of 
15 a f irst/embodiment of the color comparison circuitry of 
FIGURE^ 3; 

FIGURE 4B is a detailed functional block diagram of 
a second embodiment of the color comparison circuitry of 
FIGURE^ 3; and 

20 FIGURE 5 is a detailed functional block diagram of 

a selected one of the video window position control 
circuits depicted in FIGURE 3. 
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DESCRIPTION OF THE INVENTION 

FIGURE 1 is a high level functional block diagram of 
a multi-media processing and display system 100 operable 
to process and simultaneously display on a single display 
5 screen both graphics and video data according to the 
principles of the present invention. Display system 100 
includes a central processing unit (CPU) 101 which 
controls the overall operation of system 100 and 
generates graphics data defining graphics images to be 

10 displayed. CPU 101 communicates with the remainder of 
the system discussed below via a local bus 103. System 
100 also includes a real-time video data source 104. A 
real time video stream may be presented to the system VGA 
controller 105 in one of two ways. First, video data 

15 source 104 may be coupled to local bus 103 and a video 
data stream introduced through dual aperture addresses. 
In this case, video source 104 will directly address the 
system frame buffer 107. Second, video source 104 may be 
coupled directly to VGA controller 105 via a dedicated 

20 bus 109 or "video port." In this instance, VGA 
controller 105 generates the required addresses into 
frame buffer 107. Real-time video source 104 may be, for 
example, a CD ROM unit, a laser disk unit, a videotape 
unit, television cable outlet or other video data source 

25 outputting video data in a YUV format. CPU 101 operates 
in conjunction with a system memory 108 which stores 
graphics and video data on a real-time basis. System 
memory 108 may be for example random access memory (RAM) , 
floppy disk, hard disk or other type of storage device. 

30 A VGA controller 105 embodying the principles of the 

present invention is also coupled to local bus 103. VGA 
controller 105 will be discussed in detail below; 
however, VGA controller 105 generally interfaces CPU 101 
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and video source 104 with a display unit 106 and a multi- 
format system frame buffer 107. Frame buffer memory 107 
provides temporary storage of the graphics and video data 
during processing prior to display on display unit 106. 
5 According to the principles of the present invention, VGA 
controller is operable in selected modes to store 
graphics and video data together in frame buffer 107 in 
their native formats. In a preferred embodiment, the 
frame buffer area is partitioned into on-screen memory 
10 and off-screen memory. Frame buffer 107 is also a 
"unified" memory in which video or graphics data can be 
stored in either the on-screen or off-screen areas. In 
the preferred embodiment, display unit 106 is a 
conventional raster scan display device and frame buffer 
15 107 is constructed from dynamic random access memory 
devices (DRAMs) . 

FIGURE 2 is a more detailed functional block diagram 
of VGA controller 105. The primary circuitry blocks of 
VGA controller 105 include video front-end video pipeline 
20 200, memory (frame buffer) control circuitry 201, 
CRT/window control circuitry 202, video window control 
\ / registers 203, video back-end pipeline 204 and graphics 



includes a CPU interface 206 for exchanging instructions 
25 and data * via a PCI or VL bus, such as local bus 103 in 
system 100, with CPU 101. A write buffer 207 and 
conventional graphics controller 208 allow CPU 101 to 
directly control data within frame buffer 107 via memory 
control circuitry 201. 
30 In the preferred embodiment of system 100, CPU 101 

can write video data and/or read and write graphics data 
to frame buffer 107 via CPU interface 206. In 
particular, CPU 101 can direct each pixel to the frame 
•buffer using one of two maps depending on whether that 




controller 105 further 
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pixel is a video pixel or a graphics pixel. In the 
preferred embodiment, each word of pixel data ("pixel") 
is associated with one of two addresses, one which 
directs interpolation of the pixel as a video pixel 
through video front-end pipeline 200 and the other which 
directs interpolation of the pixel as a graphics pixel 
through write buffer 207 and graphics controller 208. 
As a consequence, either video or graphics pixel data can 
then be input to CPU interface 206 from the PCI/VI bus 
through a single "dual aperture" port as a function of 
the selected address. 

Data which is input through the video port 211, 
address-free. In this case, video window controls 213 
generates the required addresses to either the on-screen 
memory area or the off-screen memory as a function of 
display location for the video window. In the preferred 
embodiment, window controls 213 generate addresses using 
the same video control registers 203 used to control 
retrieval of the video in the back-end pipeline (i.e., 
the screen x and y position registers 500 and 501 
discussed below in conjunction with FIGURE 5) - When data 
is being received through both the CPU interface 206 and 
the VPORT 211 simultaneously, the data is interleaved 
into memory with the two write buffers 207 and 217 
buffering the data such that neither stream is 
interrupted or forced into a wait state at the source 
component (i.e, bus 103 or video source 104). 

It should be noted at this point that frame buffer 
107 includes at least two different data areas or spaces 
to which data can be directed by the given address 
(either CPU 103 or controls 213 ge nerated^ Each space 
can simultaneously store graphics or video data depending 
on. the selected display configuration. The on-screen area 
corresponds to the display screen; each pixel rastered 
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out of a given pixel location in the on-screen area 
defines a corresponding screen pixel. The off -screen 
area is used to store data defining a window for 
selectively overlaying the data from, the on-screen 
memory, fonts and other data necessary by controller 105. 
Further, as wil l, di scussed further below, both graphics 
and video data may be rastered rrom frame buffer 107 and 
passed through video backend pipeline 204 while only 
graphics data is ever passed through graphics backend 
pipeline 205. 

According to the principles of the present 
invention, there are alternate ways of storing and 
retrieving graphics and video data from unified frame 
buffer 107. 

For example, CPU 103 may write a static graphics 
background into part of the. on-screen memory with the 
remaining "window" in the on-screen memory area filled 
with playback video data. "Playback" video data can be 
either (1) live video data input from the VPORT; (2) YUV 
(video) data written through interface 206 by CPU 103; or 
(3) true color (5:5:5, 5:6:5, or 8:8:8) RGB graphics data 
(for example animation graphics data) written in through 
either the VPORT or interface 206. Similarly, a playback 
video background and a window of graphics data may be 
written into the on-screen area. In each of these cases, 
the data is rastered out as the display is without 
overlay; the video playback data is passed through the 
video playback Hat- a i <=; p assed thr j&uahu the video backend 
pipeline 204 as a function of display position by 
controls 202 and the graphics data passed through the 
graphics backend pipeline 250. 

Windows of data retrieved from the off-screen memory 
can be retrieved and used to occlude a portion of the 
data being rastered out of the on-screen memory. For 
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example, a window of playback data can be stored in the 
off-screen memory and a frame of static graphics data 
(either true color data or indices to CLUT 234) stored in 
the on-screen memory. In this case, the static graphics 
5 are rastered out of the on-screen memory without 
interruption and passed through the graphics backend 
pipeline 205. The window of data in the off-screen memory 
is rastered out only when the display position for the 
window has been reached by the display raster and is 
10 passed through video backend pipeline 204. As discussed 
below, data from the video backend pipeline 204 can then 
be used to selectively occlude (overlay) the data being 
output from the graphics backend pipeline 205. A 
window of static graphics data (true color or indices to 
the CLUT 234 ) can be stored irT of f -screen memory and used 
of overlay playback video from the on-screen memory, the 
playback video data is passed through the video" oickend 
pipeline 204 and the window of static graphics data is 
passed through the graphics backend pipeline 205. 
20 Bit block transfer (BitBLT) circuitry 209 is 

provided to allow blocks of graphics data within frame 
buffer 107 to be transferred, such as when a window of 
graphics data is moved on the display screen by a mouse. 
Digital-to-analog converter (DAC) circuitry 210 provides 
25 the requisite analog signals for driving display 106 in 
response to the receipt of either video data from video 
back-end pipeline 204 or graphics data from back-end pipe 
line 208 . 

In implementing the operations discussed above, 
30 video front-end pipeline 200 can receive data from two 
mutually exclusive input paths. First, in the "playback 
mode," playback (non-real time) data may be received via 
the PCI bus through CPU inrerface 206. Second, in the 
"overlay emulation mode" either real-time or playback 
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video may be received through the video port interface 
211 (in system 100 video port interface 211 is coupled to 
bus 109 when real-time data is being received) . The 
selection of video from the PCI bus or video from video 
5 port interface 211 is controlled by a multiplexer 212 
under the control of bits stored in a video front-end 
pipeline control register within video control registers 

203. In the playback mode, either CPU 101 or a PCI bus 
master controlling the PCI bus provides the frame buffer 

10 addresses allowing video front-end pipeline 200 to map 
data into the frame buffer separate and apart from the 
graphics data. In the overlay emulation mode, overlay 
input window controls 213 receives framing signals such 
as VSYNC and HSYNC, tracks these sync signals with 
15 counters to determine the start of each new frame and 
each new line, generates the required addresses for the 
real-time video to the frame buffer space using video 
window position data received from window controls 222 
(as discussed above, in the preferred embodiment, video 
20 data is always retrieved from either the on-screen on 
off-screen memory and passed through video back-end 
pipeline 204 as a function of display position) and thus 
the position data from controls 222 is used to both write 
data to memory and retrieve data therefrom) . In general, 
overlay input video control windows are controlled by the 
same registers which control the back-end video pipeline 

204, although the requisite counters and comparators are 
located internal to overlay input video control circuitry 
213. 

30 Video front-end pipeline 200 also includes encoding 

circuitry 214 that is operable to truncate 16-bit YUV 422 
data into an 8-bit format and then pack four such 8-bit 
encoded words into a single 32-bit word which is then 
written into the video frame buffer space of frame buffer 
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105. Conversion circuitry 215 is operable to convert RGB 
555 data received from either the CPU interface 206 and 
' „ the PCI bus or VPORT I/F 211 into YCrCb (YUV) data prior 

to encoding by encoding circuitry 214. Conversion 
5 circuitry 215 allows graphics data (for example in a 
5:5:5 or 5:6:5 format) to be introduced through the VPORT 
or graphics data to be converted, packed and stored in a 
YUV format in the off-screen memory space by CPU 101. 
For a more complete description of encoder 214 and the 

10 associated decoder 225 of video pipeline 204, reference 
is now made to incorporated copending coassigned 
application Serial No. 08/223, 845.. The selection and 
control of the encoding circuitry 214 and conversion 
circuitry 215 is implemented through multiplexing 

15 circuitries 212 and 216, each of which are controlled by 
bits in the video control registers. Finally, video 
front-end pipeline 200 includes a write buffer/FIFO 217 

\ which in one embodiment acts as a write buffer and in an 
alternate embodiment acts as a FIFO for the video backend 
0 1 pipeline 204. In embodiments where buffer 217 acts as a 
^ write buffer for then Y, zooming on the bac k-end, as 
discussed below is by replication. In embodiments where 
buffer 217 operates as a FIFO, then the VPORT and front 
and end color conversion by converter circuitry 215 are 
25 not used'for writing data to frame buffer 107. 

Memory control circuitry 201 includes an arbiter 218 
and a memory interface 219. Arbiter 218 prioritizes and 
sequences requests for access to frame buffer 107 
received from video front-end pipeline 200, graphics 
30 controller 208 and bit block transfer circuitry 209. 
Arbiter 218 further sequences each of these requests with 
the refresh of the display screen of display 106 under 
the control of CRT controller 202. Memory interface 219 
controls the exchange of addresses, data, and control 
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signals (such as RAS, CAS and read/write enable) to and 
from frame buffer 107. 

CRT control/video window control circuitry 202 
includes the CRT controller 220, window arbiter 221, and 
video display window controls 222. CRT controller 202 
controls the refresh of the screen of display 106 and in 
particular the rastering of data from frame buffer 107 to 
display unit 107 through DAC 210. In the preferred 
embodiment, CRT controller 220, through arbiter 218 and 
memory interface 219, maintains a constant stream of 
graphics data into graphics backend pipeline 205 from 
memory; video or playback graphics data is rastered out 
only when a window has been reached by the display raster 
as determined by display position controls of window 
controls 222 (see FIGURES 3 and 5 and accompanying text) 
and CRT controller 220. As will be discussed in further 
detail below, the display of windows within the display 
according to the principles of the present invention is 
controlled in part by circuitry 202. 

Video back-end, pipeline 204 receives a window of 
graphics video data defining a display window from the 
on-screen or off-screen spaces in frame buffer 107 
through a pair of first-in/first-out memories 223 and 217 
(in embodiments where buffer 217 is acting as FIFO B) . 
In the preferred embodiment, each FIFO receives the data 
for every other display line of data being generated for 
display on the display screen. For example, for a pair 
of adjacent lines n-1 and n+1 in memory (although not 
necessarily adjacent on the display) for the display 
window, FIFO 223 receives the data defining window 
display line n-1 while FIFO 224 receives the data 
defining window display line n+1. When buffer 217 is 
used as FIFO B, writes through video front end pipeline 
200 are made through write buffer I 207 and multiplexer 
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235. Alternatively, if buffer 217 is used as write 
buffer II, then FIFO B is not implemented and only a 
single stream is processed by video ba ck-end pipeline 204 
(no Y interpolation is performed and Y**expansion is by 
replication) . As will be discussed further below 
(assuming both FIFO A and FIFO B are being used) , one or 
more display lines, which falls between line n-1 and line 
n+1, may be selectively generated by interpolation. 
Decoder circuitry 225 receives two 32-bit packed words 
(as encoded by encoder 214), one from each adjacent scan 
line in memory, from FIFOs 223 and 217. Each 32-bit 
word, which represents four YCrCb pixels, is expanded and 
error diffused by decoder 225 into four 16-bit YCrCb 
pixels. In modes where video data is stored in the frame 
buffer in standard 555 RGB or 16 YCrCb data formats, 
decoder block 225 is bypassed. 

Back-end video pipeline 204 further includes a Y 
nterpdlator 226 and X interpolator 227. In the 
preferred embodiment, during Y zooming (expansion) Y 
interpolator 226 accepts two vertically adjacent 16-bit 
RGB or YCrCb pixels from the decoder 225 and calculates 
one or more resampled output pixels using a four subpixel 
granularity. X interpolator 227 during X zooming 
(expansion) accepts horizontally adjacent pixels from the 
Y interpolator 226 and calculates one or more resampled 
output pixels using a four subpixel granularity. For 
data expansion using line replication, Y interpolator 226 
is bypassed. Y interpolator 226 and X interpolator 227 
allow for the resizing of a video display window being 
generated from one to four times. 

The output of X interpolator 227 is passed to a 
color converter 228 which converts the YCrCb data into 
RGB data for delivery to output multiplexer 304. To 
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reiterate, if graphics data is passed through video 
pipeline converter 228 is not used. - — 

S ack-end video circuitry 204 further includes 
pipeline control circuitry 229, overlay control circuitry 



circuitry 239 controls the reading of data from video 
FIFOs 223 and 217, controls the generation of 
interpolation coefficients for use by X and Y 
interpolators 226 and 227 to resize the video window 
being pipelined, and times the transfer of data through 
the pipeline. Overlay control circuitry 230 along with 
control circuitry 202, controls the output of data 
through output multiplexer 231, including the overlay of 
the video window over the graphics data output through 
the graphics back-end^ pipeline 205. A pixel doubler is 
provided to double the number of pixels being generated 
such that a 1280 x 1024 display can be driven. 

Graphics back-end , pipeline 205 includes a 
first-in/first-out memory 232, attribute controller 233, 
and color look-up table 234. Each 32-bit word output 
from graphics FIFO 232 is serialized into either 8-bit, 
16-bit or 24-bit words. The 8-bit words, typically 
composed of an ASCII code and an attribute code, are sent 
to attribute controller 233. When 16-bit and 24-bit 
words, which are typically color data, are serialized, 
those words are sent di rec ti y^ gver lay controls 230. 
Attribute controller 233 performs such tasks as blinking 
and underlining operations in text modes. The eight bits 
output from attribute controller 233 are pseudo-color 
pixels used to index CLUT 234. CLUT 234 preferably 
outputs 24-bit words of pixel data to output multiplexer 
231 with each index. When video data is being pipelined 
through graphics backend pipeline 205 from the on-screen 
memory, CLUT 234 is bypassed. 



230 and output multiplexer 231. 



Pipeline control 



t 
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The eight bit- pseudo-color pixels are output from 
attribute controller 233 are also sent to overlay 
controls 230. In the preferred embodiment, data is 
continuously pipelined from on-screen memory through 
graphics back-end pipeline 205 to the inputs of output 
multiplexer 23l . " Window data from off-screen memory 
however is only retrieved from memory and pipelined 
through video backend pipeline 204 when a window is being 
displayed. In other words, when a window has been 
10 reached, as determined by control bits set by CPU 101 in 
VW control registers 222, video window display controls 
222 generate addresses to retrieve the corresponding data 
from the off-screen memory space of frame buffer 107 . 
Preferably, video FIFOs 223 and 224 are filled before the 
15 raster scan actually reaches the display window such that 
the initial pixel data is available immediately once the 
window has been reached. In order to insure that 
graphics memory data continues to be provided to graphics 
back-end pipeline 205, video window display controls 222 
20 "steal" page cycles between page accesses to the graphics 
memory. It should be noted that once the window has been 
reached the frequency of cycles used to retrieve window 
data increases over the number used to fill the video 
FIFOs when outside a window. When the frequency of 
25 window page accesses increases, video window display 
controls 222/arbiter 221 preferably "steal" cycles from 
page cycles being used to write data into the frame 
buffer . 

FIGURE 3 is a more detailed functional block diagram 
30 emphasizing the circuitry controlling the overlay of data 
from graphics pipeline 205 with window data from video 
pipeline 204. As discussed briefly above, the inputs to 
output multiplexer 231 are data from video back-end^, 
pipeline 204 (pixel doubler 237), 16 or 24-bit color data 




• 
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directly from graphics bac k-end pipeline 205 serializer 
236 and 24-bit color data from the color look-up table 
234. The output of data to DAC 210 through output 
multiplexer 231 is controlled by a latch 301 clocked by 
the video clock (VCLK) . The remaining circuitry shown in 
FIGURE 3, which will be discussed in further detail 
below, provide the necessary control signals to the 
control inputs of output multiplexer 231 to select 
between the video and graphics pipelines . 

The graphics pseudo-pixels output from attribute 
controller 233 and the 16-bit or 24-bit graphics or video 
data output directly from serializer 236 are provided to 
the inputs of color comparison circuitry 302. Also input 
to color comparison circui t 302 are 16 or 24-bit overlay 
color key bits stored in overlay color key register 303. 
Overlay color key register 303 resides within the address 
space of, and is loaded by, CPU 101. Depending on the 
mode, color comparison circuitry 302 compares selected 
bits from the overlay color key register 303 with either 
the 8 bits indexing look-up table 234 in the color 
look-up table mode (pseudo-color mode) or the 16-bits 
(24-bits in the alternate embodiment) passed directly 
from serializer 236. It should be noted that in the 
illustrated embodiment, overlay color key register 303 
holds 24-bits of overlay color key bits, eight each for 
red, green, and blue /index comparisons. The specific 
overlay color key bits compared with the input graphics 
data are provided in Table I: 

MODE OVERLAY COLOR KEY BITS COMPARED 

CLUT Blue/Index 

Index <7:0> 



5:5:5 Red<4:0> 



Green<4 : 0> 



Blue<4 : 0> 
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5:6:5 Red<4:0> Green<5:0> Blue<4 : 0> 

■8:8:8 Red<7:0> Green<7 : 0> Blue<7 : 0> 

As shown in FIGURE 4A, a first embodiment of color 
comparison circuitry 303 performs the comparisons set 
5 forth in Table I as a set of XNOR operations in series 
with an AND operation. FIGURE 4A depicts first 

comparison circuitry 400 for comparing the 8-bits of 
graphics pixels received in the look-up table mode from 
attribute controller 233 with the 8-bit blue/index 
10 overlay key bits being held in overlay key register 303. 
Second comparison circuitry 401, performs the required 
\ comparisons of Table I for the 16-bit data or 24-bit 

\ I received from serializer 236 , in, either a 5:5:5, 5:6:5, 
V/ or 8:8:8 format. An overlay register 402 includes a bit 
15 loaded by CPU 101 which is used by a selector 403, 
depending on the mode, to select for output, either the 
result of the comparisons being made by comparison 
circuitry 400 in the color look-up table mode or the 
results of the comparisons being made by comparison 
circuitry 401. In the illustrated embodiment, color 
comparison circuitry 303 processes data on a pixel-by- 
pixel basis and is resynchronized with both the graphics 
back-end ^p ipeline 205 and the video back -end p ipeline 204 
by having its outputs latched to the video clock (VCLK) 
25 by latches 404. 

The output of color comparison circuitry 303 is 
passed to the "K" control input of overlay control 
multiplexer 304. The "P" control input to multiplexer 
304 is provided from pixel position comparison circuitry 
30 305. The data inputs to multiplexer 304 are coupled to 
an 8-bit overlay OP Code (OOC) register 306. The output 
of multiplexer 304 is used as one control input to output 
multiplexer 304, which along with a single bit set by CPU 
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101 into output control register 307, selects which of 
the data received at the data inputs of multiplexer 231 
will be output to DAC 210. 

Pixel position comparison circuitry 305 includes 
5 three inputs coupled respectively to video window 1 
position control circuitry 308, CRT position control 
circuitry 309 and video window 2 position control 
circuitry 310. In the illustrated embodiment, CRT 
position controller 309 is located within CRT controller 
10 220 while video window 1 position control circuitry and 
video window 2 position control circuitry 310 are located 
within video display window controls 222 (FIGURE 2) . CRT 
position control circuitry 309 includes counters which, 
track the position of the current pixel being generated 
15 for display. In the preferred embodiment, CRT position 
control circuitry 309 includes at least an x-position 
counter which tracks the generation of each pixel along 
a given display line and a y-position counter which 
tracks the generation of each display line in a screen. 
20 The x-position counter may for example count pixels by 
counting each VCLK period between horizontal 
synchronization signal (HSYNC) controlling display unit 
106. The y-position counter may for example count each 
HSYNC signal occurring between each vertical 
synchronization signal (VSYNC) controlling the screen 
generation on display unit 106. FIGURE 4B is an 

alternate embodiment of the color comparison circuitry of 
FIGURE 3. In a first mode, 8 bits a re from attribute 

controller -233 are passed througtT multiplex er t o 
comparator 406. Comparator 406 compares the received 
eight bits with an 8-bit color key in color key register 
408; when the received 8-bits equal the 8-bit key 1, the 
output of comparator 406 goes active (high) . In the 
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first mode, control signal 16BITGR is high (and 

the output of NOR gate 409 is consequently high) and an 
active output from comparator 406 is gated through AND 
gate 410. The output of AND gate 410 is passed to AND 
5 gate 411 and gated with the output from the pixel 
comparison circuitry 305. The output of AND gate 411 
goes directly to the "B" control input of selector 231 
(in this embodiment multiplexer 304 and register 306 are 
eliminated) . Thus, when the 8-bit graphics pixels output 

10 from attribute controller 233 of graphics backend 205 
matches the 8-bit color key 1 and the window has been 
reached as determined by pixel comparison circuitry 305, 
the pixel data output from video backend 204 are passed 
through selector 231. 

15 In a second mode, 16 bits are received from 

serializer 236. The eight LSBs are passed through 
multiplexer 405 to comparator 406 and the eight MSBs 

passed to comparator 407. Control signal 16 BITNG 

is set high. When the LSBs equal key 1 in color register 
20 key 408 and the 8 MSBs equal 'key 2 in color key register 
408, the outputs from comparators 406 and 407 are active 
(high) . The output of AND gate 411 then goes high when 
the output from pixel comparison circuitry 305, which is 
coupled to the "B" control input of selector 231, goes 
25 high. Thus, when the 16-bit pixel data output from 
serializer 236 of graphics backend 205 matches the 16-bit 
color key (keys 1 and 2) and a window has been reached, 
the output pixel data from video backend 204 are passed 
through selector 231. 
30 FIGURE 5 is an expanded functional block diagram of 

the video window position control circuits 308 and a 
corresponding portion of the gating of pixel position 
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compare circuitry 305. Each position control circuit 
310/312 is coupled to a screen position x-register 500 
and a screen position y-register 501, and includes a 
screen x-position counter 502, and a screen y-position 
counter 503. In the preferred embodiment, registers 500 
and 501 are located within video window control registers 
203. For the window corresponding to the given video 
window control circuitry 308 or 310, registers 500 and 
501 are loaded with a value representing the x and y 
screen position of the pixel in the upper left corner of 
that window (the starting pixel) . Screen x-register 500 
and screen y-register 501 in the preferred embodiment are 
loaded by CPU 101. The screen x-position counter 502 
counts down from the value held in screen x-register 500 
with each video clock when P is high for each display 
line and resets with each display horizontal 
synchronization signal (HSYNC) (Note that when P is high 
the CRT count matches the position count) . Screen y- 
position counter 503 counts down from the value set into 
screen y-register 501 for each horizontal sync signal 
(HSYNC) at the start of each display line and resets with 
each VSYNC at the start of each new screen (The position 
counters are allowed to count only when they match their 
perspective CRT) . The counts values in the counters of 
CRT position control circuitry 309 are compared pixel by 
pixel with the counts in screen x-position counter 502 
and screen y-position 503 of each video window position 
control circuitry 308 and 310. When both the x and y 
counts in the counters of CRT position control circuitry 
309 match the corresponding x and y counts in respective 
counters 502 and 503 of either video window control 
circuitry 308 or 310, the control signal P to multiplexer 
304 is activated. The activation of control signal P 
indicates that the raster scan on display 106 has reached 
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the position of a pixel within the window and data from 
video pipeline 205 may be painted depending on^the value 
being held in overlay OP Code (OOC) register 306 and the 
K control inputs to multiplexer 304. 

A 4-bit OP Code loaded by CPU 101 into overlay OP 
Code register 306 in conjunction with the control signals 
applied to the M P" and "K" control inputs to multiplexer 
304 control the presentation of an active (assumed high 
in the illustrated embodiment) control signal to the "B" 
control input to output multiplexer 231. The other ("A") 
input to output multiplexer 231 receives a bit from 
overlay mode register 402 (FIG. 4), as loaded by CPU 101. 
In the illustrated embodiment, the selection between the 
streams from the graphics and video backends at the 0,1,2 
inputs to output multiplexer 304 in response to the 
signals presented at the corresponding control inputs "A" 
and "B" is in accordance with Table II: 



Control Input A 


Control Input B 


Selected Stream 


0 


0 


Graphics or 
video pixels 
from graphics 
pipeline 205 


1 


0 


Graphics pixels 
from CLUT 234 at 
input 1 


0 
1 


1 
1 


Video or 
graphics from 
video backend 
204 



The OP Codes used in the illustrated embodiment, the 
effective overlay and the corresponding inputs to the 
control inputs of multiplexer 304 are listed in Table III 
(active state is assumed : 
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Overlay Op Code 
Register 309 Value 



0 



Multiplexer 307 
Control Inputs 

N/A 



Input P active 



Inputs K and P 
active 



Inputs K active 



Effect 

Pixels passed 
only from 
graphics pipeline 
205 

Paint pixels from 
video backend 
pipeline 204 only 
inside video 
window VDW n 

Paint from video 
backend pipeline 
204 window VDW n 
when color key 
matches 

Paint from video 
backend pipeline 
204 if color key 
matches 



In the illustrated embodiment, if a Oh is written 
into OOC register 306 by CPU 101, only pixels from 
graphics pipeline 205 are pipelined through multiplexer 
304. In this case any signals applied to the P and K 
control inputs to multiplexer 304 have no effect (i.e., 
will not result in a high output from multiplexer 304) . 
In the illustrated embodiment, if an Ah is written into 
OOC register 306, pixels from video pipeline 204 will be 
passed to DAC 210 only when pixel position comparison 
circuitry 305 determines that the raster scan has reached 
a pixel in the windowand hence the control signal going 
to the P input of multiplexer 304 has been activated. If 
on the other hand, an 8h is written into OOC register 
306, data is passed through output multiplexer 231 to DAC 
210 when pixel position comparison circuitry 305 
determines that the raster scan has reached a pixel on 
the display screen within the window and color compare 
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circuitry 302 has determined that the incoming data from 
graphics pipeline 205 matches the overlay color key held 
in overlay color key register 303. In this case, the 
data from video pipeline 204 is passed to DAC 210 when 
5 both the P and the K inputs to multiplexer 304 are 
active. Finally, when an OpCode of C is programmed into 
OOC register 306, data from video pipeline 204 is passed 
if the incoming data from graphics pipeline 205 matches 
the overlay color key held in overlay color key register 
10 303. In this case, the activation of the K control input 
activate the output of multiplexer 304 to switch the 
input of multiplexer 231 to pass the corresponding video 
pixels . 

Display control circuits embodying the principles of 
15 the present invention have substantial advantages over 
the prior art. In particular, output control circuits 
built in accordance with the principles of the present 
invention allow for the flexible display of both graphics 
and video on the same screen. In particular, pixel 
20 position comparison circuitry 305 along with video window 
position control circuits 308 and 310 and CRT position 
control circuitry 309 allow fo r a _ c^ne or more windows 
from off-screen memory to be generated in specific areas 
of a display screen to the exclusion of any 
25 simultaneously generated data from on-screen memory. 
Further, color comparison circuitry 302 operating in 
conjunction with an overlay color key written into 
overlay color key register 303 allows window data to be 
presented on the display screen, to the exclusion of any 
30 concurrently generated graphics data, without the need 
for precise x- and y-position data for the window. 
Finally, the use of the graphics data from the graphics 
pipeline 205 to control the output overlay provides 
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additional advantages since the video data can be subject 
to graininess and noise. 

Although the present invention and its advantages 
have been described in detail, it should be understood 
that various changes, substitutions and alteration s an be 
made herein without departing from the spirit and scope 
of the invention as defined by the appended claims. 




